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(57) ABSTRACT 

A technique, associated system and program code, for retriev- 
ing depth information about at least one surface of an object, 
such as an anatomical feature. Core features include: project- 
ing a composite image comprising a plurality of modulated 
structured light patterns, at the anatomical feature; capturing 
an image reflected from the surface; and recovering pattern 
information from the reflected image, for each of the modu- 
lated structured light patterns. Pattern information is prefer- 
ably recovered for each modulated structured light pattern 
used to create the composite, by performing a demodulation 
of the reflected image. Reconstruction of the surface can be 
accomplished by using depth information from the recovered 
patterns to produce a depth map/mapping thereof. Each signal 
waveform used for the modulation of a respective structured 
light pattern, is distinct from each of the other signal wave- 
forms used for the modulation of other structured light pat- 
terns of a composite image; these signal waveforms may be 
selected from suitable types in any combination of distinct 
signal waveforms, provided the waveforms used are uncorre- 
lated with respect to each other. The depth map/mapping to be 
utilized in a host of applications, for example: displaying a 
3-D view of the object; virtual reality user- interaction inter- 
face with a computerized device; face — or other animal fea- 
ture or inanimate object — recognition and comparison tech- 
niques for security or identification purposes; and 3-D video 
teleconferencing/telecollaboration. 

29 Claims, 7 Drawing Sheets 
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FIG. 13 
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Modulate each of a plurality of first, second, etc, 
structured light patterns with a distinct signal waveform 


Project a composite image comprising the 
modulated first structured light patterns, at an object 
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Project a second, third, etc. composite image at the object, each V~M36 
composite comprising respective modulated patterns (alternatively) 


Capture an image reflected from one or more surface of the object, 
for each of the composite images projected at the object 
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Recover pattern information from each reflected image, for |/~M40 
each of the original modulated structured light patterns, by ' 
performing a demodulation of the reflected image 
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Reconstruct, or otherwise characterize features of, the surface: 

first, retrieve phase information from recovered 

alternative/direct route 

patterns-produce phase map/distribution 
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using depth information from the recovered 
patterns-produce a depth map/mapping 
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Retrieve additional information and/or characterize depth map, as necessary: 

1 

Compare with records of other 
depth maps for recognition of a 
face, components, or other feature 

SIB 

obtain orientation or positional data 
from either depth map or phase 
information/map for use in operation 
of user-interaction interface 



Perform selected additional operations, if any 


146b 


no— ►) continue 
















US 7,844,079 B2 


1 

SYSTEM AND TECHNIQUE FOR 
RETRIEVING DEPTH INFORMATION 
ABOUT A SURFACE BY PROJECTING A 
COMPOSITE IMAGE OF MODULATED 
LIGHT PATTERNS 

This application is a continuation of U.S. utility patent 
application Ser. No. 10/444,033 filed 21 May 2003 now U.S. 
Pat. No. 7,440,590 which was granted priority to U.S. provi- 
sional patent application 60/382,202 filed 21 May 2002, both 
of which were filed on behalf of the assignee hereof for the 
applicants. 

The invention disclosed herein was made with United 
States government support awarded, in part, by the NASA 
EPSCoR Cooperative agreement NCC5-222 and proposal 
number N02R04 D through Western Kentucky University, 
and in part by support awarded by National Science Founda- 
tion under grant EPS-9874764. Accordingly, the U.S. Gov- 
ernment has certain rights in this invention. 

BACKGROUND OF THE INVENTION 
Field of the Invention 

In general, the present invention relates to structured light 
systems that utilize multi-pattern techniques, whereby mul- 
tiple proj ected structured light patterns are used to reconstruct 
an image. More-particularly, the instant invention is directed 
to a technique and system that employs at least one camera 
and one projector used in concert for retrieving depth infor- 
mation about at least one surface of an object employing 
traditional, and new, structured light pattern projections. The 
unique method, system, and program code of the invention 
incorporate the projection of a composite image comprising a 
plurality of modulated structured light patterns, at an object. 
Recovery of initial pattern information from an image 
reflected from the object is done for each of the modulated 
structured light patterns, in a manner that preserves depth 
information within each recovered pattern. From the depth 
information, the surface can be reconstructed by way of pro- 
ducing a depth map/mapping thereof. 

While many of the known multi-pattern techniques address 
problems of depth ambiguity, invariance to surface reflec- 
tance variations, and depth accuracy, when using the tradi- 
tional technique of projecting multiple patterns to reconstruct 
an image in motion, it takes an inordinately long time to 
capture all the necessary information for reconstruction. The 
consequence of this is that these known multi-pattern tech- 
niques become quite sensitive to object movement during the 
projection capture process. To address this problem, prior 
attempts have been made by others to increase the projection/ 
capture rate. Unfortunately, as the projection/capture rate is 
increased the capture duration decreases the amount of light 
captured. Synchronization becomes more critical and the 
SNR of the capture images decreases. More-particularly the 
applicants’ invention is directed to utilizing the spatial dimen- 
sion that is orthogonal (herein, simply “orthogonal dimen- 
sion”) to the depth distortion (i.e., “phase dimension”) to 
modulate and combine multiple pattern projections into a 
single composite pattern. Applicants’ hereby disclose a new 
technique that offers a flexible way to systematically combine 
multi-patterns of structured light obtained using any selected 
technique, into a single composite pattern that can be con- 
tinuously projected — thus, minimizing delay between pattern 
projections — providing for real-time 3D video imaging. Pref- 
erably, the multi-patterns are obtained using techniques that 
benefit from current solutions to depth ambiguity, invariance 
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to surface reflectance variations, and depth inaccuracy. Using 
a composite pattern projected according to the invention, 
applicants’ have identified a way to identify position and 
perform face recognition. 

5 The composite pattern technique of the invention enables a 

wide range of multi -image methods to be utilized on moving 
objects. By modulating multiple structured light images into 
separate “channels” along the orthogonal dimension, band- 
width is utilized — just as in communications systems — re- 
io suiting in a resolution tradeoff along the orthogonal dimen- 
sion. To fiirther refine the applicants’ unique technique, of 
particular focus is any limitation(s) imposed by current state 
of the lithographic technology used to make the composite 
pattern, the number of pixel units along the orthogonal 
1 5 dimension of current camera technology, and monochromatic 
aberrations traditionally encountered using available optics 
technology. 

DISCUSSION OF BACKGROUND 
20 TECHNOLOGY 

Classic Structured Light Projection 

Typically, light structures function by projecting a pre- 
25 defined source onto a surface and then mapping the corre- 
sponding distortion in the light structure to surface features. 
Structured light measurement techniques provide a useful 
means by which data about a 3D surface may be acquired 
without contacting the surface, and used for a wide variety of 
30 engineering and industrial applications. 

Structured light is the projection of a light pattern (plane, 
grid, or more complex shape) at a known angle onto an object. 
‘Light’ and ‘images’ thereof, as used herein, includes electro- 
magnetic (EM) energy radiated throughout the EM spectrum, 
35 and more preferably, within the spirit and scope of the inven- 
tion, while the full EM spectrum is available for carrying out 
the invention, the focus is on EM emission(s) which fall 
within an extended range from the ultraviolet category (wave- 
lengths from -1 80 nm) through near-infrared (NIR) category 
40 (wavelengths from -2000 nm). In the event EM radiation 
falling outside the range of -180 nm through -2000 nm is 
used according to contemplated features of the invention, 
equipment capable of projection and capture of the radiation 
must be selected to accommodate that radiation. For example, 
45 if acoustic frequencies are employed, associated acoustic 
transmitter and receiving units must be selected for proj ection 
and capture of the composite ‘image’ information for pro- 
cessing to recover the structured Tight’ patterns initially 
modulated to compose the composite that is projected/trans- 
50 mitted at the surface of an object. 

One type of traditional light patterning often used in pro- 
cess control machine vision is generated by fanning out a light 
beam into a ‘sheet’ of light. When the sheet of light intersects 
with an object, a bright line of light can be seen on the surface 
55 of the object. By viewing this line of light with a camera 
oriented at an angle, the observed distortions in the line can be 
translated into height variations. Scanning the object with the 
light constructs 3D information about the shape of the object, 
often referred to as active tri angulation. This is the basic 
60 principle behind depth perception for machines, or 3D 
machine vision. Since structured lighting can be used to 
determine the shape of an object in machine vision applica- 
tions, as well as help recognize and locate an object in other 
environments. Structured lighting has proven useful in 
65 assembly lines implementing process or quality control, by 
offering an automatic means to check for alignment and com- 
ponent breakage or condition, for example. Stocker Yale, Inc. 
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distributes an off-the-shelf LASIRIS™ laser projector useful 
for process control in manufacturing lines to carry out com- 
ponent inspection and alignment. 

As pointed out in Daley and Hassebrook, “ Improved Light 
Sectioning Resolution by Optimized Thresholding” SPIE 
Proceedings, 2909, 151-160 (November 1996), traditional 
approaches to structured light illumination include light 
stripe projection onto a surface topology and then analyzing 
the lateral displacements of the reflected pattern to recon- 
struct the surface topology. While a single spatial frequency 
of a light stripe pattern may be used to illuminate a relatively 
flat surface, in the case of rough surfaces, the surface topology 
is preferably encoded with a sequence of light stripe patterns 
with successively higher spatial frequencies. Either way, 
maximum resolution is limited by the maximum spatial fre- 
quency used. As spatial frequency increases, the projection 
system’s blurring function causes the light stripes to be 
coupled thereby decreasing the SNR of the reflected image. 

Correctly relating distortion to surface features is funda- 
mental to structured light techniques. Discussion of how this 
has traditionally been addressed can be found in the technical 
manuscript of Raymond C. Daley, entitled “ Design , Imple- 
mentation and Analysis of Structured Light Systems ,” 
(1997) — hereafter “Daley (1997)” — submitted in fulfillment 
of a Masters of Science degree, and labeled ATTACHMENT 
B as incorporated by reference in applicants’ pending provi- 
sional application for background technical discussion. One 
fundamental geometric mechanism for creating the distortion 
is depicted in FIG. 1 of Daley (1997): it is the triangulation 
which consists of a source incident on a surface at a known 
angle, displaced laterally relative to a fixed viewing perspec- 
tive, by the surface. Lateral displacement Ax can be found by 
viewing or measuring the location of the reflected source, and 
surface height deviation Ah is determinable by trigonometry. 

FIG. 6 of Daley (1997) illustrates an example of a simu- 
lated single-stripe surface scan where the surface has been 
modeled as a 3D Gaussian pulse and the stripes modeled as a 
light plane intersecting the surface at a fixed projection angle. 
Only one light plane is depicted in FIG. 6, Daley (1997), 
representing the stripe projected onto the surface, while mul- 
tiple intersections between the light strip and the surface are 
shown to represent a scan across the surface. From Daley 
(1997) FIG. 6 one can see how the stripes map to the surface 
contour. For each stripe image received, only one slice or 
section of the surface can be reconstructed. In order to recon- 
struct the entire surface, it is necessary to move the surface or 
projector and receiver such that the stripe is projected and 
viewed at each point on the surface to be measured. This is 
quite a cumbersome process. 

FIG. 7 of Daley (1997) shows how the stripe projections 
from the simulated scan in FIG. 6, might appear if viewed by 
the receiver positioned directly above the surface. This 2D 
image represents lateral offsets which correspond to displace- 
ment Ax from Daley (1997) FIG. 1, thereby allowing surface 
height to be determined at the scanned intervals (pre-regis- 
tration/calibration required). Daley (1997) discusses several 
of the difficulties encountered in converting a 2D stripe image 
into 3D data. A technique called multi-stripe projection 
extends single- stripe system by scene illumination with mul- 
tiple slits or stripes that may be spatially modulated. Daley 
(1997) beginning at page 21, and pages 73-79 discusses two 
major components to structured light systems, namely, the 
projection and imaging subsystems designed to transmit and 
capture reflected light while maintaining or enhancing infor- 
mation in the light structure. 

FIGS. 5 and 6 of Hassebrook, Daley, and Chimitt, “ Appli- 
cation of Communication Theory to High Speed Structured 
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Light Illumination ,” Edited by Harding and Svetkoff, SPIE 
Proceedings, 3204(15), 102-113 (October 1997), concerns a 
structured light technique advancement made by at least one 
of the applicants hereof Hassebrook, et al. (October 1997) 

5 draws an analogy between the proj ection of a structured light 
pattern to encoding a surface with a spatial carrier “image” 
analogous to a carrier signal from communications theory. 
Structured light systems were treated as wide bandwidth par- 
allel communications channels. Surface characteristics (in- 
to eluding topology) act to modulate the carrier image. Known 

SLM (spatial light modulator) devices allow one to program 
the encoding of the surface with a sequence of encoded pat- 
terns, as depicted in FIG. 5 and FIG. 6 of Hassebrook, et al. 
(October 1997). 

15 In an effort to improve structured light systems (to which 
communications analogies were applied) with an eye toward 
optimizing spatial frequency while maintaining a fixed range 
resolution of the light- stripes, a technique was presented to 
enhance lateral resolution by multiplexing the light structure 
20 to produce interlaced encoded images, see Daley and Hasse- 
brook, “ Channel capacity model of binary encoded struc- 
tured light-stripe illumination ,” Applied Optics, 37(17), 
3689-3696, June (1998). FIG. 8 therefrom illustrates a first set 
of eight different stripe patterns used to encode the object; the 
25 stripe pattern having the highest spatial frequency shown at 
the upper left in FIG. 8 (spatial period of 8 pixels/cycle). A 
second set of stripe-encoded images was captured identical to 
that shown in FIG. 8 except that all the stripe patterns are 
offset 14-wavelength of the highest stripe frequency (14*8, or 
30 2 pixels/cycle). As explained by Daley and Hassebrook, June 
(1998) the two sets of images are binarized. An encoded 
image, one for each set, is generated as a weighted sum of the 
eight binary images. To interlace the two encoded images, 
stripe edge locations were determined. By combining the low 
35 bit images from both sets, a four-level encoded image was 
obtained, as illustrated in FIG. 9 of Daley and Hassebrook, 
June (1998). 

As mentioned, known structured-light illumination tech- 
niques used for automated inspection and measuring surface 
40 topologies, are cumbersome. Traditional 3D acquisition 
devices use a single scanning laser stripe scanned progres- 
sively over the surface of the target object, placing a burden on 
the object to remain static and a burden on data acquisition to 
capture all the stripe images. In an attempt to reduce compu- 
45 tational burdens of scanning and processing each scan posi- 
tion of the laser stripe, certain methods have been devised to 
project and process structured-light patterns, such as multi- 
stripe and sinusoidal fringe patterns, that illuminate the entire 
target surface at the same time. Specifically, one known tech- 
50 nique focused at addressing the ambiguity and the albedo 
problems is to encode the surface repeatedly with multiple 
light striped patterns with variable spatial frequencies. These 
known multi-stripe patterns suffer from drawbacks such as 
introducing ambiguities in the surface reconstruction around 
55 surface discontinuities, overly sensitive to surface reflectance 
variations (i.e., albedo), and/or they suffer from lower lateral 
resolution caused by the required spacing between stripes. 
These known systems are simply not suitable for real-time 
imaging. 

60 Thus, and according to the invention, a very unique 
approach is outlined herein as supported by rigorous math- 
ematical and engineering analyses performed by the appli- 
cants. Applicants have discovered a structured-light pattern- 
ing that allows, with a single projected composite image, the 
65 measuring of surface topologies that addresses issues of 
ambiguities, higher accuracy, and less sensitivity to albedo 
variations. 
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SUMMARY OF THE INVENTION 

It is a primary object of this invention to provide a tech- 
nique and system for retrieving depth information about at 
least one surface of an obj ect employing traditional, and new, 5 
structured light pattern projections. The unique method, sys- 
tem, and program code disclosed and supported herein incor- 
porate the projection of a composite image comprising a 
plurality of modulated structured light patterns, at an object; 
and recovery of initial pattern information from an image 10 
reflected from the object, for each of the modulated structured 
light patterns, in a manner that preserves depth information 
within each recovered pattern. From the depth information, 
the surface can be reconstructed by way of producing a depth 
map/mapping thereof. The nature of applicants’ new tech- 15 
nique permits the depth map/mapping to be utilized in a host 
of applications, for example: displaying a 3D view of the 
object; virtual reality type human/user-interaction interface 
with a computerized device; face — or other animal feature, or 
any feature of an inanimate object, distinguishable by surface 20 
characteristics — recognition and comparison techniques for 
security or identification purposes; and 3D video teleconfer- 
encing/telecollaboration. 

As one will appreciate, the unique features supported 
hereby may provide advantages of display clarity, versatility 25 
in applications/functionalities, speed, efficiency, overall sys- 
tem cost -reduction, permitting reliable display information to 
be communicated to a viewer. Specific advantages of provid- 
ing the new method and associated system include, without 
limitation: 30 

(a) Ease of operability and design simplicity — -The inven- 
tion provides an ability to automatically recover depth infor- 
mation about an object under observation with a camera, in a 
manner useful and readily adaptable to new real-time 3D 
imaging, 3D object analysis (e.g., face recognition, compo- 35 
nent identification and alignment, etc., whereby unique 
topologies are identified and matched) as well as adaptable to 

a wide variety of other new commercial/industrial and 
research/scientific applications. 

(b) 3D image process design flexibility and versatility of 40 
use — The technique of the invention can be tailored for use 
with a wide variety of EM emissions/light patterns and pat- 
terning projected (for example, from ultraviolet through NIR 
ranges) and captured using suitable associated imaging com- 
ponents, subsystems, and computing capabilities to carry out 45 
the applications. The processing may be carried out in digital 

or analog equivalents. 

(c) Manufacturability and overall system cost reduction — 
The unique technique and system of the invention can be 
tailored to current, as well as any under development or 50 
yet-to-be-developed, imaging systems providing a cost-ef- 
fective means by which systems can be upgraded, or sold as 
OEM. 

(d) Time required to acquire and generate depth informa- 
tion and associated maps of a surface can be reduced accord- 55 
ing to the invention, making projection and capture of the 
light/EM radiation suitable for the real-time applications con- 
templated hereby. 

Briefly described, once again, the invention includes a 
method, as well as associated system and computer execut- 60 
able program code on a computer readable storage medium, 
for retrieving depth information about at least one surface of 
an object. The method, system and code incorporate(s) the 
steps of: projecting a composite image comprising a plurality 
of modulated structured light patterns, at the object; capturing 65 
an image reflected from the surface; and recovering pattern 
information from the reflected image, for each of the modu- 
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lated structured light patterns. Pattern information is prefer- 
ably recovered for each of the modulated structured light 
patterns used to create the composite, by performing a 
demodulation of the reflected image. Reconstruction of the 
surface can be accomplished by using depth information from 
the recovered patterns to produce a depth map/mapping 
thereof. Preferably, each signal waveform used for the modu- 
lation of a respective structured light pattern, is distinct from 
each of the other signal waveforms used for the modulation of 
other structured light patterns used to create a composite 
image; these signal waveforms may be selected from suitable 
types including: frequency shifted waveforms, phase shifted 
waveforms, random noise waveforms, and encoded wave- 
forms. Any combination of distinct signal waveforms may be 
employed for modulation of a set of structured light patterns 
used to create a composite image, provided the waveforms 
used are uncorrelated with respect to each other. The tech- 
nique may be carried out using digital processing or analog 
electronic circuitry. 

As one will appreciate, there are many further distinguish- 
ing features of the system and technique, and associated 
program code, of the invention. A surface reconstructed by 
way of a depth map, may simply be displayed as a 3D gray- 
scale or color still image or video imaging (boxes 144, FIG. 
13). Additional applications of the depth mapping include 
(boxes 146 and 148, FIG. 13): comparing the depth map with 
a plurality of stored depth map records of an animal feature, 
such as a face or hand, or of any other feature such as the 
distinguishing surface characteristics of a component being 
assembled, and so on, for purposes of identification or recog- 
nition of the object; or one may obtain orientation or posi- 
tional data about the object from the depth map, such as roll, 
pitch, and yaw angle data, or positional data (coordinates for 
points or arrays) for use in connection with operation of a 
user-interaction interface for a wide variety of computerized 
devices. The composite image can be made up of structured 
patterns projected as electromagnetic energy radiated 
throughout the electromagnetic spectrum, and more-prefer- 
ably within a wavelength range from about 1 80 nm through 
2000 nm, inclusive. The step of capturing a reflected image 
can include the receiving and recording of pixel intensity and 
pixel wavelength (e.g., using RGB — red, green, blue — color 
display capable video) of the energy reflected from the sur- 
face. Components suitable for projection and capture of the 
composite image are selected according to the energy/wave- 
length range of the light patterns employed. 

Further unique to the invention is an opportunity to, after 
capturing the first reflected image, project a second composite 
image comprising a second plurality of modulated structured 
light patterns, at the object so that second pattern information 
from a second reflected image can be recovered for each of the 
second plurality of modulated structured light patterns. This 
feature of time-sequencing composites, an example of which 
is depicted by the isometrics in FIGS. 9A-9B, offers the 
capability to produce a depth map of the surface(s) of the 
object using pattern information recovered from two, or 
more, projected composite images. Such a depth map can 
reconstruct several surfaces of the object from reflected 
imaged information received sequentially in time. 

In another aspect of the invention a second composite 
image comprising a second plurality of modulated structured 
light patterns can be projected, at the object. Here, preferably, 
the first and second composite images are projected using, 
respectively, electromagnetic energy radiated within a first 
and a second wavelength range, whereby the first and second 
wavelength ranges have no overlap making the composites 
‘bandwidth limited’. An example of this alternative is 
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depicted by the isometric drawing in FIG. 7. Such a technique 
provides for a means by which, whether done simultaneously 
or sequentially, two or more composite images can be pro- 
jected at the object to gather additional surface information — 
providing the capability to recover additional depth informa- 
tion and, thus, produce more detailed and/or larger depth 
mappings. 

In yet another aspect of the invention, a second composite 
image comprising a second plurality of modulated structured 
light patterns, is projected at the object; however, the projec- 
tion of the second composite image is oriented such that 
orthogonal and phase dimensions of the second composite 
image are unaligned from the orthogonal and phase dimen- 
sions of the first projected composite image. An example of 
this alternative is depicted by the isometric drawing in FIG. 8 . 
Once again, as in the case of the non-overlapping frequency 
range alternative above, this aspect of the invention provides 
an option for simultaneous or sequential projection: two or 
more composite images can be projected at the object to 
gather additional surface information. 

The system for retrieving depth information about at least 
one surface of an object under observation by at least one 
camera, as characterized includes: at least one projector/ 
transmitter adapted for projecting at least one composite 
image at the object; the camera/capture unit adapted for cap- 
turing an image being reflected from the surface; and a pro- 
cessor adapted for recovering pattern information from the 
reflected image, for each modulated structured light pattern, 
the depth information retrievable from the recovered pattern 
information. A second, third, and so on, projector/transmitter 
adapted for respectively projecting a second, third, and so on, 
composite image accordingly comprising a second, third, and 
so on, plurality of modulated structured light patterns, at the 
object, may be added to carry out the additional distinguish- 
ing features discussed above. Similarly a second, third, and so 
on, camera/capture unit adapted for respectively capturing a 
second, third, and so on, reflected image may be added to pair 
with additional projectors employed to project second, third, 
and so on, composites. The addition of transmitter-receiving 
unit pairs to carry out composite projection and associated 
reflected image capture, can be programmed to perform the 
unique alternative aspects of the invention discussed through- 
out, including: time-sequencing of composite images, non- 
overlapping structured light frequency ranges (bandwidth 
limited composites), and composite unalignment techniques. 

The computer executable program code on a computer 
readable storage medium for retrieving depth information 
about at least one surface of an object under observation by at 
least one camera, as characterized includes: a first program 
sub -code for projecting a composite image comprising a plu- 
rality of modulated structured light patterns, at the object; a 
second program sub-code for recovering pattern information 
from an imaged reflected from the surface, for each of the 
plurality of modulated structured light patterns; and a third 
program sub -code for reconstructing the surface using the 
recovered pattern information. The program code preferably 
also includes an initial program sub-code for modulating with 
a distinct signal waveform, each of a respective structured 
light pattern to obtain the plurality of modulated structured 
light patterns. The second program sub-code can comprise 
instructions for performing a demodulation of the reflected 
image to produce recovered pattern information for each of 
the plurality of modulated structured light patterns; and the 
third program sub-code can comprise instructions for produc- 
ing a depth map of the surface using the recovered pattern 
information. 
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Furthermore, additional program sub -code can be 
employed for the comparing of the depth map with a plurality 
of stored depth map records of an animal feature for purposes 
of identification of the object; and/or additional program sub- 
5 code can be employed for obtaining orientation data about the 
object from the depth map for use as explained above; and/or 
program sub-code can be employed for carrying out the pro- 
jection of second, third, and so on, composite images utilizing 
the unique composite time-sequencing, non-overlapping 
10 structured light frequencies (bandwidth limited composites), 
or composite unalignment techniques explained herein; and/ 
or program sub-code may be employed for the recovery of 
respective pattern information for each additional composite 
so projected, to generate even move-complex depth maps-all 
15 supported and contemplated hereby. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For purposes of illustrating the innovative nature plus the 
20 flexibility of design and versatility of the preferred technique 
and system, and associated program code, supported and 
disclosed hereby, the invention will be better appreciated by 
reviewing the accompanying drawings (in which like numer- 
als, if included, designate like parts). One can appreciate the 
25 many features that distinguish the instant invention from 
known or traditional techniques. The drawings and any 
attachments hereto have been included to communicate the 
features of the innovative system and associated technique of 
the invention as well as the rigorous analysis performed by the 
30 applicants by way of example, only, and are in no way 
intended to unduly limit the disclosure hereof. 

FIG. 1 schematically depicts a projector-camera, or other 
suitable transmitter-receiver, system 10 for carrying out the 
technique of the invention, having projector/transmitter P x for 
35 projecting a composite image onto object surface(s) S and 
camera/capture unit C : capturing the image that is accord- 
ingly reflected from S. 

FIG. 2 is a pictorial representing one embodiment depict- 
ing data/information/signal patterns combinable using modu- 
40 lation technique(s) to create a composite image 20 according 
to the invention. 

FIG. 3 graphically depicts the frequency spectrum of an 
example captured image reflected from surface S for the four 
channel composite pattern represented in FIG. 2. Note that 
45 modulated pattern information is recoverable from the 
reflected image. 

FIG. 4 is a high-level block diagram of a process suitable 
for use to recover the pattern information for each modulated 
structured light pattern. The image is demodulated by suitable 
50 digital or equivalent analog circuit signal processing to 
recover each individual pattern within the captured compos- 
ite, which now contains surface depth information. 

FIGS. 5A-5D are pictorials representing various stages of 
carrying out an embodiment of the invention according to 
55 EXAMPLE 1C as follows: FIG. 5A shows the projection 
pattern on a reference plane; FIG. 5B shows the recovered 
reference phase mapping for projection of FIG. 5A; FIG. 5C 
shows the captured image of the object plane whereby a 
half-circular step has been placed atop the reference plane of 
60 FIG. 5A; and FIG. 5D shows the recovered phase map of the 
object plane of FIG. 5C; 

FIG. 6 is a depth mapping of the FIG. 5C object scene 
having been reconstructed by calculation in a pixel -wise fash- 
ion according to Eqn. (4). 

65 FIGS. 7, 8, and 9A-9B are high-level isometric pictorials, 
each of which depict a multi-transmitter-receiving unit sys- 
tem 70, 80, 90 having for example, projectors P 1; and P 2 both 
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projecting to contribute to a respective composite image/ 
region of object surface S, according to alternative tech- 
niques: structured light frequency range/bandwidth limited 
composite technique, composite dimension-unalignment 
technique, and composite time- sequencing projection tech- 
nique. Each projector/transmitter is paired with a camera, or 
other suitable image capture/receiver unit, C 1 and C 2 , posi- 
tioned for capturing reflected, respective composite images 
according to the invention. 

FIG. 10 is an isometric pictorial depicting a reconstructed 
object, here for example a human hand, along with references 
as to roll, pitch, and yaw orientations thereof. 

FIGS. 11A-11D are each pictorials depicting different ori- 
entations of an object, here for example a human hand, onto 
which a composite image is projected (shown at left side of 
each FIG.); with a respective 3D mapping reconstructed from 
captured reflected images, and shown for each orientation of 
the hand-object on the right side of each FIG. 

FIGS. 12A-12B are pictorials depicting, respectively, a 
captured image of a human hand oriented with a pitch angle of 
-55 degrees onto which a composite has been projected, and 
a corresponding range image reconstructed therefrom. 

FIG. 13 is a flow diagram depicting details of a method 130 
for retrieving depth information about at least one surface of 
an object according to the invention. Illustrated are core, as 
well as further distinguishing, features for projecting com- 
posite images and/or capturing reflected images such as those 
represented and depicted in FIGS. 2, 5A-5D, 6, 10, 11A-11D, 
and 12A-12B, using features of FIGS. 1, 3, 4, 7, 8, 9A-9B. 

DETAILED DESCRIPTION OF EMBODIMENTS 
DEPICTED IN DRAWINGS 

In connection with discussing the figures, occasional ref- 
erence will be made to the FIG. 13 flow diagram which details 
core and further distinguishing features of a technique of the 
invention at 130. FIG. 1 schematically depicts a projector- 
camera system 10 for carrying out the technique of the inven- 
tion, having projector P x for projecting a composite image 
onto object surface(s) S and camera Cj^ capturing the image 
that is accordingly reflected from S. FIG. 2 is a pictorial 
representing one embodiment depicting data/ signal/informa - 
tion patterns combinable using modulation technique(s) to 
create the composite image 20. As shown, multiple structured 
light patterns are modulated (box 132, FIG. 13) utilizing 
suitable carrier patterns 16a-16d and combined 17 into a 
composite pattern 20. Traditional multi-frame patterns may 
be used produced by any suitable technique such as Phase 
Measuring Profilometry (PMP) patterns 18a-18d, Linearly 
Coded Profilometry (LCP), as well as any traditional or new 
technique capable of generating a plurality of patterns, pref- 
erably at least from and throughout ultraviolet, visible, and 
NIR ranges, that may be modulated and combined through 
processing (computerized unit 14) and projected P : as a com- 
posite 20 onto one or more selected surfaces S of a target 
object (box 134, FIG. 13). Additional composites may, like- 
wise, be projected (box 136, FIG. 13). 

The individual patterns ISa-lSd of FIG. 2 are spatially 
modulated 16a-16a( 17 along the orthogonal dimension, per- 
pendicular to the phase dimension — each of which are refer- 
enced, here. While many suitable patterns may be modulated 
according to the invention, for simplicity of discussion, the 
focus of discussion herein is on coding and decoding proce- 
dures of composite patterns for the PMP technique, by way of 
example. Employing system 10, a ‘single’ frame of a com- 
posite PMP pattern is formed 20 and projected through Pi to 
the target object S. The reflected image is received and cap- 
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tured by camera C x (box 138, FIG. 13) and decoded employ- 
ing suitable processing, computerized unit 12, to retrieve the 
multiple PMP frames from the captured image (box 140, FIG. 
13) so that the phase distribution distorted by the object depth 
5 can be calculated (box 142, especially 142a, FIG. 13). Pattern 
information is preferably recovered for each of the modulated 
structured light patterns 18a -18d used to create the composite 
20, by performing a demodulation of the reflected image (box 
140, FIG. 13). The depth, or depth map/mapping, of the 
1 0 obj ect can accordingly be reconstructed out of the phase (box 
142, especially 1426, FIG. 13), relying on traditional struc- 
tured light techniques for doing so while utilizing classic 
demodulation practices of communications theory, tailored to 
1 5 the type of structured light patterning used to create the com- 
posite. Once again, for the embodiments depicted in FIGS. 
2-6 the composite patterning technique used to recover the 
four structures from the captured image while preserving 
depth information about the surface S, was the PMP tech- 
20 nique, by way of example, only. 

Preferably, modulation of each structured light pattern 
(box 132, FIG. 13) used to project a given composite image, 
is done with a distinct signal waveform. The signal wave- 
forms may be selected from many suitable types, including 
25 those listed here: frequency shifted waveforms, phase shifted 
waveforms, random noise waveforms, and encoded wave- 
forms. Any combination of distinct signal waveforms may be 
employed for modulation of a set of structured light patterns 
used to create a given composite image, provided the wave- 
30 forms used are uncorrelated with respect to each other. 

In connection with the following discussion concerning 
derivation and analysis by way of example, employing PMP, 
FIGS. 1-6 will occasionally be referenced. FIG. 3 graphically 
depicts at 30, along the orthogonal dimension, the frequency 
35 spectrum of an example captured image reflected from sur- 
face S for the four channel composite pattern represented in 
FIG. 2. Note that modulated pattern information is recover- 
able from the reflected image. FIG. 4 is a high-level block 
diagram of a process 40 suitable for use to recover the pattern 
40 information for each modulated structured light pattern; for 
reference see also FIG. 13 at 140 and 142. The captured image 
42 is filtered and demodulated (represented at 44) to recover 
each individual pattern containing the desired depth informa- 
tion. Finally phase information is retrieved 46. While FIG. 4 
45 depicts components of digital processing, the technique of the 
invention is suitably carried out using analog electrical com- 
ponent equivalents, as contemplated hereby. 

EXAMPLE 1A 

50 

Derivation & Analysis Discussion — Composite 
Patterning Employing PMP 

The PMP range finding method has several advantages , it i s 
a pixel -wise calculation, resistant to ambient light, resistant to 
reflection variation and it can have as few as three frames for 
whole-field depth reconstruction. Sinusoid patterns are pro- 
jected and shifted by a factor of 2 jt/N for N times as 

60 cos (2 Jtj" ( ^y p -2jm/N) Eqn. (1) 

where A p and B^ are the projection constants and (x p ,y p ) is the 
projector coordinates. The y p dimension is in the direction of 
the depth distortion and is called the phase dimension. On the 
65 other hand, x p dimension is perpendicular to the phase dimen- 
sion, so it has been coined herein, the orthogonal dimension. 
Thus, frequency of the sinusoid wave is in the phase 
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directi on/dimension. The subscript n represents the phase 
shift index and n=l , 2, . . . , N, where N is the total number of 
phase shifts. 

The reflected intensity images from the object surface after 
successive projections are 

I n (x,y)=a(x,y)- [A+B co s (2 jtJ" ^+( j) (x,y) -2%n/N ) J Eqn. (2) 

where (x,y) are the image coordinates and a(x,y) is the reflec- 
tance variation or the albedo. The pixel -wise phase distortion 
(|)(x,y) of the sinusoid wave corresponds to the object surface 
depth. The value of 4>(x,y) is determined from the captured 
patterns by 


(j>{x, y) = arctan 


I n (x, y)sm(7jm/N) 


21 /„(*, y)cos(2jm/N) 

,n= 1 


Eqn. (3) 


Since albedo, a(x,y), is cancelled in this calculation, one can 
see that depth through this approach is independent of the 
albedo. 

When calibrating the range finding system, the phase map 
of the reference plane 4>^ (x,y) is pre-calculated from projec- 
tions on the reference plane (labeled R in FIG. 1). The depth 
of the object surface S with respect to the reference plane R is 
readily obtained using simple geometric relationships. As 
shown in FIG. 1, the distance between the projector lens 
center, O , to the camera lens center, O c , is d. Both the 
projector and the projector-camera plane are a distance L 
from the reference plane. The height, h, of the object at a given 
point A can be determined by 


BC-(L/d) Eqn. (4) 

(1 +BC}d) 


where BC is proportional to the difference between the phase 
at point B, and the phase at point C, c|) c , as 

Eqn. (5) 

The constant |3, as well as other geometric parameters, L and 
d, are determined during calibration procedure. The phase 
value calculated from Eqn. (3) is wrapped in the range value 
of (-Tt, jc] independent of the frequencies in phase direction. 
Phase unwrapping procedure retrieves the non-ambiguous 
phase value out of the wrapped phase. With relatively higher 
frequencies in phase direction, the range data have higher 
signal-to -noise-ratio (SNR) after non-ambiguous phase 
unwrapping. 

In order to combine multiple patterns into one single com- 
posite image 20, each individual pattern is modulated along 
orthogonal direction with a distinct carrier frequency and 
then summed together as depicted by FIG. 2. Therefore, each 
channel in the composite image along the orthogonal direc- 
tion represents the individual pattern used in PMP for the 
phase calculation. Similar to the patterns projected in multi- 
frame approach as in Eqn. (1), the image patterns to be modu- 
lated are governed by 

IJ p =c+<x>s(2itf^y p -2jtn/N) Eqn. (6) 

A constant c is used here to offset \ p to be non-negative 
values. One can appreciate that negative signal values will 
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cause an unusable demodulation employing this amplitude 
modulation (AM) based demodulation method, as discussed 
later. The signal patterns are then multiplied with cosine wave 
with distinct carrier frequencies along the orthogonal direc- 
5 tion. The composite pattern accumulates each channel such 
that 


N Eqn. (7) 

i p = a” 


where f n p are the carrier frequencies along the orthogonal 
15 direction and n is the shift index from 1 to N. The projection 
constants A p and B^ are carefully calculated as 


20 


25 


A p = I mm - B p ■ mini \ I p ■ cos(2 nf p x p ) 


B p = 


(Imax-Imin) 



N n 1 


maxi 

2 In - cos (2nf n xP)j 

~ 


( N „ 


min 

| 21 I P -COS(2 7tf n X p ) 

11 


Eqn. (8) 


Eqn. (9) 


so that the projection intensity range of the composite pattern 
30 falls into \l min , I max ]. In order to increase the SNR, B^ should 
reach its maximum value allowed and therefore, [I /mw , I max \ 
should match the intensity capacity of the proj ector to retrieve 
optimal depth information. 

In this example, orthogonal modulation frequencies f n p 
35 have been selected to be evenly distributed and away from 
zero frequency. This modulation is analogous to classic AM 
modulation. No patterns have been modulated in the “DC” or 
baseband channel. Although the bandwidth of the composite 
pattern is degraded by losing baseband channel, the modula- 
40 tion pattern is less sensitive to ambient light. Ideally, the 
reflected composite pattern image on the target object surface 
captured by the camera is 

45 f n \ Eqn. (10) 

Icp(x, y) = a(x, ;y)j A + B • ^ /' (x, y) ■ cos(2 xf n x) l 


V n {x, y) = c + cos(2 nf 0 p + (f>(x, y) - 2nnjN). Eqn. (11) 


where a(x,y) is the albedo and (|)(x,y) is the distorted phase as 
in Eqn. (2). The actual carrier frequencies f n in the camera 
55 view may be different from the f/ due to perspective distor- 
tion between the projector P x and the camera C v To make the 
modulation frequency f n as independent as possible of the 
topology of the object surface on each orthogonal line, the 
camera and projector may be carefully aligned to share 
60 approximately the same world coordinates both in orthogonal 
direction and depth direction. If the orthogonal and phase 
axes of the camera and projector fields have a relative rotation 
between them, the orthogonal carrier modulation of the pro- 
jector may leak into the phase component captured by the 
65 camera. 

Since projector P : and camera C l digitally sample the pro- 
jection pattern and captured image, the detection of the high 
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frequency carrier wave and the recovery procedure rely 
heavily on the intensity and the spatial resolution of the pro- 
jector and camera system. Appropriate carrier frequency, f n p , 
has to be carefully assigned. Selection of the carrier fre- 
quency S/ is dependent on the projector and camera quality, 
as well as the experimental setup. Preferably, while adjacent 
carrier f/ should be spread out as much as possible to mini- 
mize the channel leakage, due to limitations of spatial and 
intensity resolution carrier f / should be confined to a certain 
range for reliable depth recovery. 

Reflected images are processed, here, as 1-D raster signals 
where each line along the orthogonal dimension is an inde- 
pendent signal vector. The received orthogonal spectrum 30 
for four composite pattern channels, in a typical signal vector, 
is illustrated in FIG. 3. Here, the four carrier frequencies are 
evenly distributed and are separated from the ambient light 
reflection at baseband. The captured image is processed, as a 
set of 1-D signal vectors, by band-pass filters to separate out 
each channel. To achieve uniform filtering for the channels, 
the band-pass filters are centered at f n and are all derived from 
the same low-pass Butterworth filter design; that is to say, 
they all have the uniform passband span and are symmetric at 
f„. The Butterworth filter is used in this stage for smoother 
transition and minimal side-lobe ripple effect. The order of 
the Butterworth filter is carefully selected to reduce the cross- 
talk between channels. A compromise is made between side- 
lobe effects and cross-talk in order to obtain acceptable recon- 
struction performance. Cutoff frequencies for each band are 
designed such that 


This is low pass filtered by h i/? '(x) with a cutoff of f„ such that 


5 


g(x, y) = (/f p (x, y)f *h' LP (x) = 


(I'nix, y)f 
2 


Eqn. (16) 


The modulated image is recovered by square rooting Eqn. 
(16) such that 

//(^)=^5i^JJ=V2-[(/ n BP (^)) 2 ’VW] Eqn. (17) 

Due to the involvement of the square operation in the 
demodulation process, I^^y) has to be non-negative. Thus, 
it is effectively an AM based modulation technique which 
recovers the PMP pattern as the positive envelope. The 
demodulation procedure is depicted at 40 in FIG. 4. The 
recovered images I n R (x,y) represent the individual patterns in 
traditional PMP and are used to retrieve the depth of the 
measured object. 

The range data with respect to the reference plane can then 
be determined according to Eqn. (4) and (5). While one can 
appreciate from Eqn. (13), leakage error between orthogonal 
channels occurs when the measured object surface has sig- 
nificant variation of albedo or depth in the orthogonal direc- 
tion, reconstructed depth in the phase direction is resistant to 
depth discontinuity and albedo variation. 

EXAMPLE IB 

Addendum Summary to Discussion and Analysis 
employing PMP 


f n C =^(fn-l+fn) 


Eqn. (12) 


where n=l, 2, 3, . . . , N and f o =0 which is the baseband 
channel. The orthogonal signal vectors after 1-D band-pass 
filtering are 

In BP (x,y)=Icp(x,y) *h £P n (x)~I n '(x,yycos(2jif n x) Eqn. (13) 

where * is the convolution operator and hgp”(x) is the band- 
pass filter along orthogonal direction centered at frequency 
f„. The baseband image I w ’(x,y) is assumed to be band limited 
along the orthogonal dimension with a bandwidth less than or 
equal to the filter h Br n (x) bandwidth. 

The filtered images have to be demodulated to retrieve each 
individual pattern, l w '(x,y). In the demodulation process, two 
critical factors are considered: First, the perspective distor- 
tion causes the depth dependent variation of orthogonal car- 
rier frequencies. Second, in carrying out the technique, the 
cosine carrier wave on each orthogonal line has an unknown 
phase shift. That is, considering the perspective distortion, the 
image after band-pass filtering can be based on Eqn. (13), 
such that 

I n BP (x,y) =/„ 1 (x,y ) ■ co s (2 Jt (f „+8f )x+60) . Eqn. (14) 

where has small variation 8 j and 80 is the unknown phase 
shift. By squaring both sides of Eqn. (14) we have 


i jBPf \\2 ^2 1 + cos ( 47 t (/„ + 6f)x + 269) Eqn. (15) 

(/„(*. y)) = y)f 


As mentioned, one core feature of the technique of the 
invention is the projection of a composite 20, FIG. 2 of modu- 
lated patterns. Assuming scale and offset of the composite 
function has a transmittance between 0 and 1, the composite 
image may be represented by the following expression: 


N Expression [I] 

S C p(x, y) = J] fn(x)<Pn(y) 

n= 1 


45 The structured light pattern set is represented with a 1-D 
function f„(x) where x is along the phase dimension. The 
modulating function is (|) w (y) where y is along the orthogonal 
dimension. The index is n=l, 2, . . . N. Drawing from com- 
munications concepts to use AM frequency division multi - 
50 plexing, encode the N patterns. Let 

<j)„0)=^(l+cos(2jtf^„);)) Expression [II] 

where A is a scaling constant and j y n is the orthogonal fre- 
55 quency. 

The well known multi-pattern PMP technique employed 
here to provide the structured light patterns for use in the 
composite pattern, uses phase shifted sinusoids as the projec- 
tion patterns defined by 

60 > f ?3 (x)=l +cos(2jtf_ x x:+2jt«/7V) Expression [III] 

where f x is the spatial frequency and the phase is uniformly 
shifted from 0 to 2 jt as a function of n. Many different types 
of multi-patterning techniques, including the many optical 
65 profilometry techniques, may be employed to profile a sur- 
face. The image patterns of PMP are shown in FIG. 2 at 
18a-18d where f x is selected as explained above. 
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After the composite image is reflected off the object sur- 
face, the depth distorts the f n (x) functions f„(g(x w , y w , ZJ) 
where g(.) is the distortion of x by the surface world coordi- 
nate {x w , y w , Z w } as seen by the camera The reflected 
image is demodulated to get the original patterns which con- 
tain the distorted depth, from which the range is calculated. In 
the process of demodulation, the reflected image may be 
element-wise multiplied by the orthogonal images. Once 
again, the original ‘now-distorted’ patterns can be recovered 
by demodulation along the orthogonal dimension with the 
distortion intact (for reference, see FIG. 4). The non-ambigu- 
ous depth can be reconstructed from the demodulated pat- 
terns using conventional reconstruction. 

FIGS. 5A-5D and FIG. 6 are pictorials representing various 
stages of carrying out an embodiment of the invention accord- 
ing to EXAMPLE 1C, below. FIG. 5 A shows the projection 
pattern on a reference plane. FIG. 5B shows the recovered 
reference phase mapping for projection of FIG. 5A. FIG. 5C 
shows the captured image of the object plane whereby a 
half-circular step with diameter of 300 mm and thickness of 
85 mm has been placed atop the reference plane of FIG. 5A. 
FIG. 5D shows the recovered phase map of object plane of 
FIG. 5C. FIG. 6 is a depth mapping 60 of the FIG. 5C object 
scene having been reconstructed by calculation in a pixel- 
wise fashion according to Eqn. (4). See, also, FIG. 13 (box 
142) as explained above: The phase distribution distorted by 
the object depth can be calculated (for example, 142a); with 
the depth, or depth map/mapping, of the object accordingly 
reconstructed out of the phase (for example, 1426). 

FIGS. 7, 8, and 9A-9B are high-level isometric pictorials, 
each of which depict a multi -transmitter-receiving unit sys- 
tem 70, 80, 90 having for example, projectors P l5 and P 2 both 
projecting to contribute to a respective composite image/ 
region of object surface S, according to alternative tech- 
niques: structured light frequency range/bandwidth limited 
composite technique, composite dimension-unalignment 
technique, and composite time- sequencing projection tech- 
nique. Each projector/transmitter is paired with a camera, or 
other suitable image capture/receiver unit, C l and C 2 , posi- 
tioned for capturing reflected, respective composite images 
according to the invention. Turning first, to FIGS. 9A-9B 
which depict a time-sequencing composite projection tech- 
nique, one can appreciate that this aspect provides an oppor- 
tunity to, after capturing the first reflected image of the pro- 
jected composite at 91, projecting a second composite image 
92 comprising a second plurality of modulated structured 
light patterns, at the object 966 so that second pattern infor- 
mation from a second reflected image can be recovered for 
each of the second plurality of modulated structured light 
patterns. This feature of time- sequencing composites, shown 
in FIG. 9 A at time T1 as 90a and in FIG. 9B at time T2 as 906, 
offers the capability to produce a depth map of the surface(s) 
of the obj ect using pattern information recovered from two, or 
more, projected composite images. Such a depth map can 
reconstruct several surfaces of the object from reflected 
imaged information received sequentially in time. 

FIG. 7 depicts a technique whereby structured light fre- 
quency is range, or bandwidth, limited. Here, preferably, the 
first and second composite images are projected at object 76 
(composite 71 projected by P x and composite 72 projected by 
P 2 ) using electromagnetic energy radiated within respective 
first and a second wavelength ranges, whereby the first and 
second wavelength ranges have no overlap making the com- 
posites ‘bandwidth limited’. Such a technique provides for a 
means by which, whether done simultaneously or sequen- 
tially, two or more composite images 71, 72 can be projected 
at the object 76 to gather additional surface information — 
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providing the capability to recover additional depth informa- 
tion and, thus, produce more detailed and/or larger depth 
mappings. 

FIG. 8 depicts, yet another aspect coined composite dimen- 
5 sion-unalignment. Here, a second composite image 82 com- 
prising a second plurality of modulated structured light pat- 
terns, is projected at the object 86: however, the projection of 
the second composite image 82 (from projector P 2 ) is ori- 
ented such that orthogonal and phase dimensions of the sec- 
10 ond composite image are unaligned from the orthogonal and 
phase dimensions of the first projected composite image 81 
(from PJ. See, also, FIG. 2 for reference. Once again, as in 
the case of the non-overlapping frequency range alternative 
above, this aspect of the invention provides an option for 
15 simultaneous or sequential projection: two or more compos- 
ite images can be projected at an object to gather additional 
surface information. 

FIG. 10 is an isometric pictorial depicting a reconstructed 
object 100, here for example a human hand, along with ref- 
20 erences as to roll, pitch, and yaw orientations thereof. Accord- 
ing to the invention, positional coordinate or array data may 
be used alone or with orientation (146a, 1466, FIG. 13). 
FIGS. 11A-11D are each pictorials depicting different orien- 
tations of an object, here for example a human hand, onto 
25 which a composite image is projected (shown at left side of 
each FIG.); with a respective 3D mapping reconstructed from 
captured reflected images (box 146, FIG. 13), and shown for 
each orientation of the hand-object on the right side of each 
FIG. FIGS. 12A-12B are pictorials depicting, respectively, a 
30 captured image of a human hand oriented with a pitch angle of 
-55 degrees onto which a composite has been projected, and 
a corresponding range image reconstructed therefrom. In 
FIG. 12B, the range map represents an intensity coded depth 
map where the whiter the pixel is, the closer the correspond- 
35 ing world position is along the z-axis (see FIG. 10 for refer- 
ence). As detailed within boxes 142, 146 and explained fur- 
ther throughout: phase information 142a from recovered 
patterns can be used to produce depth maps/mappings 
1426 — whether or not the depth mappings are displayed 144; 
40 and alternatively, phase information retrieved from recovered 
patterns can be used ‘directly’ to obtain orientation or posi- 
tional data (1466), useful in operation of any of a number of 
user-interaction interfaces. Likewise, depth map information 
may be ‘directly’ used for recognition of an anatomical fea- 
45 ture, e.g., face, head, hand, etc., by comparing prior data 
records stored of like-kind (146a). For example, a depth map 
of a face undergoing recognition may be compared with 
hundreds of thousands or millions (depending on storage 
size) of records of face depth maps produced earlier (146a). 
50 By way of background, only, in connection with the picto- 
rials of FIGS. 10-12: two known types of input devices com- 
monly used to capture hand movement include a glove-based 
approach and a computer vision based approach. The glove- 
based devices are relatively accurate and fast in position 
55 tracking. However, they are not suitable for applications 
where the mechanical devices may impede hand motions. 
The vision-based approach does not require gloves but it 
suffers from computational complexity and latency. Certain 
of the vision approaches are composed of multiple cameras 
60 and are built upon stereovision techniques. Other techniques 
are based on two-dimensional (2D) information which are not 
robust enough for human computer interaction. Using the 3D 
information depth information recovered according to the 
invention, a hand or other movable feature may be more 
65 efficiently segmented. Since a world coordinate system may 
be established for the camera scene in a calibrated projector- 
camera system, depth information can be obtained using the 
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composite pattern projection of the invention without corrup- 
tion by object motion. The latency of the interface input is not 
limited by the projecting speed, which would be the case if 
employing a traditional multiple-pattern structured light tech- 
nique, but instead depends on 3D processing speed for a 
one-shot frame. 

The following are core steps utilized in the process to 
produce the pictorials of the hand shown at 100 in FIG. 10, as 
further illustrated in FIGS. 11A-11D: 

(1) Calculate the 3D data from the original captured image 
according to the invention, generally at 130 FIG. 13; and 
threshold out the background and the shadows. 

(2) Form a 2D hand shape according to the captured image 
and correlate a hand shaped spatial filter with each image to 
locate the hand center in the 2D image. 

(3) Set a rectangular region with fixed length around the 
hand center. The centroids along columns of the hand area 
within the region are calculated. A straight line is generated to 
fit the centroids in a least squares sense. The yaw angle is the 
angle in degrees between the straight line and a horizontal line 
(reference FIG. 10). 

(4) Look up the 3D world coordinates corresponding to the 
rectangular section. The pitch angle is calculated based on the 
depth values along the fit line from step (3) above, while the 
roll angle is calculated along the line perpendicular to the fit 
line. 

(5) Smooth the location coordinates and the rotation angles 
and transfer them to the virtual hand program to manipulate 
the virtual hand. 

As explained above, FIG. 13 is a flow diagram depicting 
details of a method 1 3 0 for retrieving depth information about 
at least one surface of an object according to the invention. 
Illustrated are core, as well as further distinguishing, features 
for projecting composite images and/or capturing reflected 
images such as those represented and depicted in FIGS. 2, 
5A-5D, 6, 10, 11A-11D, and 12A-12B, using features of 
FIGS. 1, 3, 4, 7, 8, 9A-9B. 

EXAMPLE 1C 

Summary of Application of Techniques Discussed in 
Examples 1A-1B 

Referencing the various FIGs throughout this EXAMPLE 
1C, the following is provided merely for purposes of further 
understanding the novel features of the invention. A Texas 
Instruments (TI) Digital Light Processor (DLP) projector, P 15 
was used with an 800x600 micro -mechanical mirror array. 
The framegrabber 12, a DT3120, grabs the image from the 
CCD monochrome camera with spatial resolution of 640x 
480 with 8 bits intensity resolution. To simplify decoding, the 
frequency across the phase direction f ^ is selected to be unit, 
or 1, frequency: No unwrapping algorithm need be imple- 
mented. In this experiment, number of patterns, N=4; carrier 
frequencies of the projector f/ were 50, 85, 120 and 155 
cycles per field of view for an orthogonal field of view width 
of 800 pixels. The corresponding received carrier frequencies 
were 33, 56, 79, and 103 cycles per field of view with a field 
of view of 640 pixels. The lowest modulation frequency was 
selected to be higher than the difference of the adjacent modu- 
lation frequencies to minimize the effect of the ambient light 
reflection. The order of the Butterworth bandpass filter was 
selected to be 7 and the width of the passband was 10 to 
reduce the cross-talk between adjacent channels. FIG. 5A 
shows the projection pattern on the reference plane and the 
recovered reference phase map is shown in FIG. 5B. To test 
sensitivity to depth variation, a half circular step with the 
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thickness of about 85 mm is placed on the top of the reference 
plane. The reflected image and the corresponding phase map 
are shown in FIG. 5C and FIG. 5D, respectively. The depths 
of the object scene were calculated pixel -wise following Eqn. 
5 (4) and are shown in FIG. 6. The demodulation procedure 
generates the edge response effects in the reconstructed 
depths. The original sharp edges of the circle from the refer- 
ence plane in the world coordinates were reconstructed with 
edge transitions between the two depth levels in the depth 
10 map due to the band limited filtering. The abrupt edges of the 
depth act as step edges in the orthogonal direction for all 
pattern channels. 

In further support of the unique features of the 3D face, or 
other feature, recognition application of the invention (box 
15 146, especially 146a, FIG. 13): 3D face recognition technique 
of the invention may be based on wavelet and Fourier trans- 
forms (particularly the spherical FFT) over the surface of a 
known object shape — for example, may be a sphere. Just as 
an audio signal can be decomposed into its spectrogram, the 
20 object shape (e.g., a sphere) will be decomposed into its 
‘spherical spectrogram’ — making the new technique effec- 
tively orientation invariant. Thus, a computationally efficient 
technique is for identifying a subject’s head position in space, 
performing a compression of the head shape, and performing 
25 face recognition, other anatomical features is contemplated 
hereby. 

While certain representative embodiments and details have 
been shown for the purpose of illustrating the invention, those 
skilled in the art will readily appreciate that various modifi- 
30 cations, whether specifically or expressly identified herein, 
may be made to these representative embodiments without 
departing from the novel teachings or scope of this technical 
disclosure. Accordingly, all such modifications are intended 
to be included within the scope of the claims. Although the 
35 commonly employed preamble phrase “comprising the steps 
of’ may be used herein in a method claim, Applicants do not 
intend to invoke 35 U.S.C. §112 ^6. Furthermore, in any 
claim that is filed herewith or hereafter, any means -plus - 
function clauses used, or later found to be present, are 
40 intended to cover at least all stmcture(s) described herein as 
performing the recited function and not only structural 
equivalents but also equivalent structures. 

What is claimed is: 

1. A method for performing recognition of an anatomical 
45 feature using depth information about a surface of the ana- 
tomical feature, comprising the steps of: 

modulating each of a plurality of structured light patterns 
using a respective carrier frequency along the orthogo- 
nal dimension; 

50 projecting a composite image comprising said plurality of 
modulated structured light patterns, at the anatomical 
feature; and 

recovering pattern information from an image reflected 
from the surface, for each of said plurality of modulated 
55 structured light patterns. 

2. The method of claim 1 wherein said step of recovering 
said pattern information comprises performing a demodula- 
tion of said reflected image, said pattern information so 
recovered from said plurality of structured light patterns com- 

60 pri sing the depth information; and further compri sing the step 
of reconstructing the surface using the depth information. 

3. The method of claim 2 wherein: each said modulated 
structured light pattern was obtained by modulation with a 
signal waveform associated with said respective carrier fre- 

65 quency; and said step of reconstructing comprises producing 
a depth map of the surface using said recovered pattern infor- 
mation. 
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4. The method of claim 3 further comprising the step of 
comparing said depth map with a plurality of stored depth 
map records of animal features of like-kind as that undergo- 
ing recognition, for purposes of identification of the anatomi- 
cal feature. 

5. The method of claim 2 wherein the anatomical feature is 
selected from the group consisting of: a face, a facial feature, 
a head shape, and a hand. 

6. The method of claim 2 wherein: said step of performing 
a demodulation comprises performing said demodulation 
along said orthogonal dimension producing a recovered pat- 
tern information for each of said plurality of modulated struc- 
tured light patterns; and said step of reconstructing comprises 
producing a depth map of the surface using said recovered 
pattern information. 

7. The method of claim 2 wherein said step of performing 
a demodulation comprises performing said demodulation 
along said orthogonal dimension; and further comprising the 
step of displaying said reconstructed surface. 

8. The method of claim 7 wherein: said projected compos- 
ite image comprises electromagnetic energy radiated within a 
wavelength range from 180 nm through 2000 nm; the image 
reflected from the surface is captured by receiving and record- 
ing pixel intensity and pixel wavelength of the energy 
reflected from the surface; and said step of displaying further 
comprises displaying the object in three-dimensional space. 

9. The method of claim 1 further comprising the steps of: 

projecting a second composite image comprising a second 

plurality of modulated structured light patterns, at the 
anatomical feature; and 

recovering second pattern information from a second 
reflected image, for each of said second plurality of 
modulated structured light patterns. 

10. A method for obtaining orientation data about an ana- 
tomical feature using information retrieved from a surface of 
the anatomical feature, comprising the steps of: 

modulating each of a plurality of structured light patterns 
using a respective carrier frequency along the orthogo- 
nal dimension; 

projecting a composite image comprising said plurality of 
modulated structured light patterns, at the anatomical 
feature; and 

recovering pattern information from an image reflected 
from the surface, for each of said plurality of modulated 
structured light patterns. 

11 . The method of claim 1 0 wherein said step of recovering 
said pattern information comprises performing a demodula- 
tion of said reflected image, said pattern information so 
recovered from said plurality of structured light patterns com- 
prising information selected from the group consisting of: 
depth information and phase information. 

12. The method of claim 10 wherein: each said modulated 
structured light pattern was obtained by modulation with a 
signal waveform associated with said respective carrier fre- 
quency; and the anatomical feature is selected from the group 
consisting of a face, a facial feature, a head shape, and a hand. 

13. The method of claim 10 wherein said orientation data 
obtained comprises data selected from the group of roll angle 
data, pitch angle data, yaw angle data, and positional data, for 
use in connection with operation of a user-interaction inter- 
face for a computerized device. 

14. The method of claim 10 further comprising the steps of: 

projecting a second composite image comprising a second 

plurality of modulated structured light patterns, at the 
anatomical feature; and 
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recovering second pattern information from a second 
reflected image, for each of said second plurality of 
modulated structured light patterns. 

15. A system for performing recognition of an anatomical 
5 feature using depth information about a surface of the ana- 
tomical feature under observation by at least one camera, the 
system comprising: 

at least one projector adapted for projecting at least one 
composite image at the anatomical feature, said com- 
io posite image comprising a plurality of modulated struc- 
tured light patterns, each having been modulated using a 
respective carrier frequency along the orthogonal 
dimension; 

the camera adapted for capturing an image being reflected 
1 5 from the surface; and 

a processor adapted for recovering pattern information 
from said reflected image, for each said plurality of 
modulated structured light patterns: the depth informa- 
tion retrievable from said recovered pattern information. 
20 16. The system of claim 15 wherein each said modulated 

structured light pattern was obtained by modulation with a 
distinct signal waveform associated with said respective car- 
rier frequency; said processor is further adapted to recover 
said pattern information by performing a demodulation along 
25 said orthogonal dimension of said reflected image; and the 
depth information is used to produce a depth map of the 
surface. 

17. The system of claim 16 wherein: each said signal wave- 
form used for said modulation of a respective structured light 

30 pattern, is distinct from each other of said signal waveforms 
used for said modulation of other structured light patterns; 
and each said distinct signal waveform is selected from the 
group consisting of frequency shifted waveforms, phase 
shifted waveforms, random noise waveforms, and encoded 
35 waveforms. 

18. The system of claim 16 wherein said processor is fur- 
ther adapted to: reconstruct the surface using the depth infor- 
mation; and compare said depth map with a plurality of stored 
depth map records of animal features of like-kind as that 

40 undergoing recognition, for purposes of identification of the 
anatomical feature. 

19. The system of claim 15 wherein the anatomical feature 
is selected from the group consisting of: a face, a facial 
feature, a head shape, and a hand. 

20. A system for obtaining orientation data about an ana- 
tomical feature using information retrieved from a surface of 
the anatomical feature under observation by at least one cam- 
era, the system comprising: 

50 at least one projector adapted for projecting at least one 
composite image at the anatomical feature, said com- 
posite image comprising a plurality of modulated struc- 
tured light patterns, each having been modulated using a 
respective carrier frequency along the orthogonal 
55 dimension; 

the camera adapted for capturing an image being reflected 
from the surface; and 

a processor adapted for recovering pattern information 
from said reflected image, for each said plurality of 
60 modulated structured light patterns. 

21. The system of claim 20 wherein each said modulated 
structured light pattern was obtained by modulation with a 
distinct signal waveform associated with said respective car- 
rier frequency; said processor is further adapted to recover 

65 said pattern information by performing a demodulation of 
said reflected image; said pattern information so recovered 
from said plurality of structured light patterns comprising 
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information selected from the group consisting of: depth 
information, and phase information. 

22. The method of claim 20 wherein: the anatomical fea- 
ture is selected from the group consisting of a face, a facial 
feature, a head shape, and a hand; and said orientation data 
obtained comprises data selected from the group of roll angle 
data, pitch angle data, yaw angle data, and positional data, for 
use in connection with operation of a user-interaction inter- 
face for a computerized device. 

23. A non- transitory computer readable storage medium 
having stored thereon, program code for performing recog- 
nition of an anatomical feature using depth information about 
a surface of the anatomical feature under observation by at 
least one camera, the program code comprising: 

a first program sub-code for modulating each of a plurality 
of structured light patterns using a respective carrier 
frequency along the orthogonal dimension and project- 
ing a composite image comprising said plurality of 
modulated structured light patterns, at the anatomical 
feature; 

a second program sub-code for recovering pattern infor- 
mation from an image reflected from the surface, for 
each said plurality of modulated structured light pat- 
terns; and 

a third program sub-code for retrieving the depth informa- 
tion from said recovered pattern information. 

24. The storage medium of claim 23 wherein: 

said first program sub-code comprises instructions for 
modulating each said plurality of structured light pat- 
terns with a distinct signal waveform associated with 
said respective carrier frequency; 

said second program sub-code comprises instructions for 
performing a demodulation of said reflected image to 
produce a recovered pattern information for each of said 
plurality of modulated structured light patterns; and 

said third program sub-code comprises instructions for 
producing a depth map of the surface using said recov- 
ered pattern information. 

25. The storage medium of claim 24 wherein the program 
code further comprises a fourth program sub-code for com- 
paring said depth map with a plurality of stored depth map 
records of animal features of like-kind as that undergoing 
recognition, for purposes of identification of the anatomical 
feature. 

26. The program code of claim 23 further comprising: 

a fourth program sub-code for projecting, after said recov- 
ering pattern information, a second composite image 
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comprising a second plurality of modulated structured 
light patterns, at the anatomical feature; and 

a fifth program sub-code for producing a depth map of the 
surface using (a) the depth information from said recov- 
ered pattern information, and (b) pattern information 
recovered for each of said second plurality of modulated 
structured light patterns. 

27. A non-transitory computer readable storage medium 
having stored thereon, program code for obtaining orientation 
data about an anatomical feature using additional information 
retrieved from a surface of the anatomical feature under 
observation by at least one camera, the program code com- 
prising: 

a first program sub-code for modulating each of a plurality 
of structured light patterns using a respective carrier 
frequency along the orthogonal dimension and project- 
ing a composite image comprising said plurality of 
modulated structured light patterns, at the anatomical 
feature; 

a second program sub -code for recovering pattern infor- 
mation from an image reflected from the surface, for 
each said plurality of modulated structured light pat- 
terns; and 

a third program sub-code for retrieving the additional 
information from said recovered pattern information, 
said additional information selected from the group con- 
sisting of: depth information, and phase information. 

28. The storage medium of claim 27 wherein the orienta- 
tion data comprises data selected from the group of roll angle 
data, pitch angle data, yaw angle data, and positional data, for 
use in connection with operation of a user-interaction inter- 
face for a computerized device. 

29. The program code of claim 27 wherein the additional 
information retrieved from said recovered pattern informa- 
tion is depth information, and further comprising: 

a fourth program sub-code for projecting, after said recov- 
ering pattern information, a second composite image 
comprising a second plurality of modulated structured 
light patterns, at the anatomical feature; and 

a fifth program sub-code for producing a depth map of the 
surface using (a) the depth information from said recov- 
ered pattern information, and (b) pattern information 
recovered for each of said second plurality of modulated 
structured light patterns. 





